Section: New Results

Large-scale image search

Aggregating local image descriptors into compact codes

Participants : Matthijs Douze, Hervé Jégou [INRIA Rennes] , Patrick Pérez [Technicolor] , Florent Perronnin [Xerox RCE] , Jorge Sánchez [Xerox RCE] , Cordelia Schmid.

In [5] we consolidate and extend earlier results for large-scale image search. Different ways of aggregating local image descriptors into a vector are compared. The Fisher vector, see Figure 1 , is shown to achieve better performance than the reference bag-of-visual words approach for any given vector dimension. Furthermore, we jointly optimize dimensionality reduction and indexing in order to obtain a precise vector comparison as well as a compact representation. The evaluation shows that the image representation can be reduced to a few dozen bytes with good search accuracy. Given such small codes, searching a 100 million image dataset takes about 250 ms on one processor core.

Figure 1. Illustration of the similarity of the Fisher vectors of local image regions despite viewpoint changes.

Searching in one billion vectors: re-rank with source coding

Participants : Laurent Amsaleg [CNRS, IRISA] , Matthijs Douze, Hervé Jégou [INRIA Rennes] , Romain Tavenard [University Rennes I] .

In this work [13] we extend our earlier work [4] . An additional level of processing is added to the product quantizer to refine the estimated distances. It consists in quantizing the difference vector between a point and the corresponding centroid. When combined with an inverted file, this gives three levels of quantization. Experiments performed on SIFT and GIST image descriptors show excellent search accuracy outperforming three state-of-the-art approaches.

Combining attributes and Fisher vectors for efficient image retrieval

Participants : Matthijs Douze, Arnau Ramisa, Cordelia Schmid.

Attributes were recently shown to give excellent results for category recognition. In [9] we demonstrate their performance in the context of image retrieval. We show that combining attributes with Fisher vectors improves performance for retrieval of particular objects as well as categories. Furthermore, we implement an efficient coding technique for compressing the combined descriptor to very small codes. Experimental results show that our approach significantly outperforms the state of the art, even for a very compact representation of 16 bytes per image. We show that attribute features combined with Fisher vectors improve the retrieval of image categories and that those features can supplement text features.

Bag-of-colors for improved image search

Participants : Matthijs Douze, Hervé Jégou [INRIA Rennes] , Christian Wengert [Kooaba] .

In [19] we investigate the use of color information when used within a state-of-the-art large scale image search system. We introduce a simple color signature generation procedure, used either to produce global or local descriptors. As a global descriptor, it outperforms several state-of-the-art color description methods, in particular the bag-of-words method based on color SIFT. As a local descriptor, our signature is used jointly with SIFT descriptors (no color) to provide complementary information.